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ABSTRACT 

It is now well accepted that working in research teams that span universities, jurisdictions and 
countries can be rewarding and economically prudent. To this end, investigators collaborate in 
the pursuit of knowledge to address human and societal problems and translate results into local 
and global contexts. This implies that investigators need to develop study instruments that are fit 
for purpose and strategically manage issues arising from geographical, linguistic and cultural 
diversity. A proven method is cognitive interviewing to pre-test the study materials to ensure 
clarity’ and relevance in the study population. This paper describes the steps taken to increase the 
methodological reliability of study instruments through the use of cognitive interviewing and 
argues this technique should be a standard step in instrument development. 
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INTRODUCTION 



here is an increasing global trend toward team-based research where relevant disciplines, decision¬ 
makers and stakeholders from industry, local communities, and professional organizations collaborate 
to generate ‘real world’ research questions and translate findings into policy and practice (Mitchell et 
ah, 2009, Golden-Biddle et al., 2003, Straus et al., 2011, Hofmeyer et ah, 2012). Research teams are now more 
likely to span faculties, universities, jurisdictions and countries and include a diversity of investigators to design 
contextually relevant instruments and translate findings into local and global contexts (Siemens and Burr 2012, 
Straus et al., 2011). In the question and answer process, the respondent must comprehend what they are being asked, 
recall the necessary information, make a judgement about the information needed to answer the question, and 
provide the answer (Collins 2003, Tourangeau 1984). However, wording, nuances and meanings in one context may 
differ in another. To date, the focus has been on standardising design protocols and data collection instruments to 
produce valid and reliable results, but response errors still occur due to ambiguity, misunderstandings and unfamiliar 
wording (Drennan 2003, Collins 2003). Researchers can implement strategies to develop and pre-test study 
instruments to ensure relevance, meaning and impact in the local context (i.e., fit for purpose) and thus, effectively 
manage issues arising from ‘geographical, linguistic and cultural diversity’ (Siemens and Burr 2012, p. 1). In the 
1980s, cognitive interview methods were developed to pre-test study materials to identify potential sources of 
response error and to ensure clarity and relevance (Willis 1999, Collins 2003, Drennan 2003, Willis et al 1991, 
Tourangeau 1984). More recently cognitive methods have been utilised as effective pre-testing techniques to address 
issues related to diversity, so can be relevant in global research (Hurst et al., 2015). 


This article is intended to contribute a new understanding about pretesting study materials to reduce the risk 
of collecting invalid and incomplete data. Our purpose is essentially twofold. First, we begin with a discussion on 
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pre-testing methods and steps to conduct cognitive interviewing. Second, we describe how we used cognitive 
interviewing to increase methodological reliability and coherence in the adaptation of USA study materials for 
replication in the UK higher education context. We conclude that cognitive methods can serve a specific purpose 
within the researcher’s tool kit and should be used with other pre-testing methods. 

PRE-TESTING METHODS 

Pre-testing methods are techniques used to pilot survey instruments to improve validity and detect word 
ambiguity that could be problematic in the formal study (Hurst et ah, 2015). Pre-testing is a broad term used to 
identify problems in survey questions and includes a range of methods and combinations that can be implemented 
pre-field (i.e., reviews by experts), or in the field (i.e., respondent de-briefing). It is argued that ‘projects that neglect 
pre-testing run the risk of later collecting invalid and incomplete data’ (Hurst et ah, 2015, p. 57). Recent advances 
in online surveys prevent respondents from moving forward in the survey if they do not answer a question, thus 
solving the problem of missing data. But respondents can still provide incorrect answers if they do not comprehend 
the wording in the question or recall accurate information. Bowden and colleagues (2002, p. 322) argue: 

claims concerning the validity, reliability and sensitivity of health-related quality of life measures tend to 
be based on the quantitative approach of psychometrics, which fails to identity when respondents: 
misinterpret questions; do not recall the information requested; or give answers that present themselves in 
a better or worse light. 

Moreover, considerable attention has been directed at the process of standardising data collection 
instruments to ensure valid and reliable results but interestingly, response errors still occur (Collins 2003, p. 230). 
Standardisation assumes that: 

• ‘all respondents understand the questions in a consistent way; 

• the questions are asking for information that respondents have and can retrieve; 

• the wording of questions provides respondents with all the necessary information they require to be able to 
answer them in a way required by the researcher; and 

• respondents always read the questions as worded’ (Collins 2003, p. 230). 

However, linguistic or lexical or problems can arise when ‘words that are familiar to one group may not be 
to another or the words may have different meanings’ (Drennan 2003, p. 58). Survey pre-testing or piloting is 
another traditional approach used by researchers to identify problematic questions that do not produce expected 
responses. But notably, researchers using this technique will not know why the question is problematic, or how to 
improve the wording. 

Therefore, there is increasing support for cognitive interviewing as a promising method to address 
potentially significant problems in research studies. This method could be used routinely to pre-test questions and 
study materials with individuals (who have similar characteristics to the study subjects) to gain crucial insights in the 
development of study instruments (Collins 2003, Drennan 2003, Willis 1999). 
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COGNITIVE INTERVIEWING 

Cognitive interviewing method is a synthesis of cognitive and social psychology and survey research 
methodology that has been increasingly utilized since the 1970s (Collins 2003, Drcnnan 2003). It is a pretesting 
technique or diagnostic tool that explores processes respondents use to understand and attribute meaning and to 
identify any problematic wording of questions (Collins 2003, Drcnnan 2003). Individuals with similar characteristics 
to the study inclusion criteria can act as respondents in this process. Content validity of the instrument can be 
determined through cognitive interviews. When utilising cognitive interviewing techniques, researchers will 
potentially gain insights to: 

• Understand how the questions or tasks are perceived and interpreted from the respondent’s perspective 
rather than that of the researcher, so in other words, in the way the researcher intended (Collins 2003; 
Drcnnan 2003); 

• Identify problems with wording and organization that could potentially lead to misunderstandings and 
missing data (Collins 2003; Drennan 2003); 

• Identify how language is used in a particular context such as a social or geographic setting, including 
cultural norms, relevance and expectation of what words and phrases actually mean which can influence 
accurate interpretation and responses (Hurst et ah, 2015, p. 57). 

• Estimate time required to complete a survey or conduct an interview (Hurst et ah, 2015). 

When associated with other reliability and validity measures, cognitive interviews can lead to the 
development of effective surveys, which is critical because non-response or non-complction of surveys is 
problematic (Drennan 2003). Researchers can utilise the following three techniques to conduct cognitive interviews: 

Think-Aloud/Read-Aloud 

Ask respondents to verbalize their thoughts and understanding aloud as they read each survey question 
aloud and then attempt to answer the question as they understand it (Hurst et ah, 2015; Collins 2003; Drennan 2003; 
Willis 1999; Ericsson & Simon 1980). This process also allows researchers to ‘decode idiomatic words’ (Hurst et 
ah, 2015, p. 58) which could be important in global research studies and ensuring translation of relevant results. 

Cognitive Verbal Probing 

Ask respondents to paraphrase the questions in their own words and to define the meanings of each word to 
explore their comprehension. Probing will identify words that could be misinterpreted or misunderstood and 
provides an opportunity to ask the respondent to rephrase the question in their own words. Probing also can explain 
how the respondent went about answering the question and recalling information (Collins 2003; Drennan 2003, 
Willis 1999, Willis et al 1991). We have successfully used the range of probe questions detailed in Table 1 to 
conduct cognitive interviews (Collins 2003, p. 235; Willis 1999, p. 6). 
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Table 1. Examples of Cognitive Probes 


Think aloud/generalVgeneral probe 2 

How did you go about answering that question? 1 

Tell me what you are thinking 1 

I noticed you hesitated before you answered - what were you thinking about? 1 

How easy or difficult did you find this question to answer? Why do you say that? 1 
How did you arrive at that answer?2 

Was that easy or hard to answer? 2 

ComprehensionVlnterpretation 2 

What does the tern X mean to you? 1 

What do you understand by X? 1 

Could you tell me in your own words what the question is asking you? 1 

How would you say that question yourself? 1 

Paraphrasing 2 

Can you repeat the question 1 just asked in your own words? 2 

Retrieval'/Recall 2 

How did you remember that? 1 

Did you have a particular time period in mind? 1 

How did you calculate your answer? 1 

How do you remember that you went to the doctor five times in the past 12 
months? 2 

Confidence judgement 1,2 

How well do you remember this? 1 

How sure of your answer are you? 1 

How sure are you that your health insurance covers drug and alcohol treatment? 2 

Response 1 

How did you feel about answering this question? 1 

Were you able to find your first answer to this question from the response option 
shown? 1 


Adapted from Collins 2003 1 , p. 235-236 and Willis 1999 2 , p. 6 


Observing 

Non-verbal behaviours of respondents such as frowning or blushing could identify problematic questions or 
words that might provoke confusion, embarrassment or anxiety in the formal study, thus risking collection of invalid 
data (Hurst et ah, 2015). Researchers can observe the respondents as they read the questions aloud and note 
examples of skipping questions, omitting words in the question, putting answers in the wrong place, and responses 
that are hesitant or indicate frustration or stress. These observations will give clues about the appropriateness of 
questions, words, topics, layout, validity and credibility of the study materials (Hurst et ah, 2015, Drennan 2003). 
Researchers can immediately ask respondents how to address any confusion. 

OUR RESEARCH STUDY 

We conducted a pre-post intervention study to explore understandings about the concept of social capital 
and how the concept could strengthen perceptions of connectedness and belonging between graduate nursing 
students studying via distance education at a university in the USA (Sheingold et al. 2014). Social capital has been 
conceptualised as bonding, bridging and linking networks with six dimensions: groups and networks; trust and 
solidarity; collective action and cooperation; information and communication; social cohesion and inclusion and 
empowerment and political action (Grootaert et ah, 2004). The study comprised: 
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1. Stage 1 Online Survey - demographic and baseline data about graduate students’ (participants) 

understandings of social capital prior to the intervention. 

2. Stage 2 Intervention - comprising the following steps: 

a) Delivery of a 20 minute scripted slide presentation to study participants (distance learners) about 
social capital and its relationship to the learning experience of graduate students in higher 
education. The slide presentation included examples and images of bonding social capital that 
exists between like-minded people; bridging social capital that exists between people with varied, 
but similar ideas; and linking social capital that exists between those with power and people who 
rely upon them (Grootaert et al 2004). 

b) In small groups, participants then discussed how they thought social capital could potentially be 
utilized by themselves as distance learners in nursing education. 

c) Participants were then asked to write down three points they had learned about social capital and 
submit on the form provided. 

3. Stage 3 Online Survey - repeat stage 1 online survey one month after the Intervention. 


Adaption of study materials for the UK context 


An opportunity arose to replicate the USA higher education study protocol in the UK higher education 
context. Researchers from the USA team and UK team collaborated in the adaptation of study materials to ensure 
relevance in the UK cultural context whilst remaining true to the theoretical concepts of bonding, bridging and 
linking social capital. As we know, linguistic or lexical or problems can arise when ‘words that are familiar to one 
group may not be to another or the words may have different meanings’ (Drennan 2003, p. 58). This was a vital 
consideration in our international higher education study. For example, words and images used to illustrate social 
capital and professional affiliations had to be familiar and have meaning for the UK students to ensure accurate 
responses. Cognitive interviewing techniques and expert review provided the framework to firstly examine the 
demographic questions in the Stage 1 Online Survey. It was critical that the wording and terms featured the higher 
education and professional categories would be familiar to respondents in the UK to ensure relevance in the local 
social and cultural context. Equivalent terms to illustrate this point are in Table 2. 


Table 2. Examples of equivalent terms 


USA protocol 

Distance education graduate students 
Department of Health and Human Services 
American Nurses Association 
American Association of Colleges of Nursing 


Adapted UK protocol - equivalent terms 
Distance learning students 
Department of Health 
Nursing and Midwifery Council 
Royal College of Nursing 


After careful discussion, each of the remaining questions in the Stage 1 Online Survey were assessed as 
relevant for the UK context and expected to elicit reliable responses. A review of the US scripted slide presentation 
was undertaken noting the theoretical intent of the accompanying images, photographs and script. A number of 
changes were made to the images, photographs and to the affiliations and professional organisations so that the 
presentation and the script matched the cultural and social context in the UK. For example, we used the logos of our 
UK professional bodies to illustrate linking social capital. These changes ensured that the presentation used 
examples were familiar to the student participants and which illustrated the theoretical concepts in ways they could 
understand. What we found was that the participants in the UK study were able to undertake the surveys and the 
research interventions, and that the wording and imagery were suitable to the context. At the same time, the meaning 
of the survey questions and the social capital theory was not lost in translation. 

Limitations of cognitive methods 


Cognitive methods has limitations, but can serve a specific purpose within the researcher’s tool kit and be 
complementary to other pre-testing methods (Collins 2003). Cognitive methods can be used to assess the relevance, 
coherence and problems with wording and framing of existing questions, images and information. However, 
researchers need to understand that this qualitative approach cannot provide quantitative information on the extent of 
the problem or the ‘size of its impact on survey estimates’ (Collins 2003, p. 236). Furthermore, cognitive methods 
cannot confirm that the revised questions and study materials are definitely better than the original version. That 
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said, this method did improve the rigor of our adapted study materials and enabled the US study to be replicated in 
the UK pilot study. 


CONCLUSION 

The process of adaptation and the methodological issues have been discussed, particularly in relation to 
establishing validity and managing lexical problems to establish whether study instruments are replicated accurately 
and questions are answered in the way the researcher intended. Content validity of study instruments and materials 
can be improved through the use of cognitive interviewing, so researchers can consider using this method to assess 
whether their study instruments are likely to generate relevant data to meet their study objectives. This method can 
also be used to review conference presentations, script and images on power point presentations and posters to 
confirm relevance and effective dissemination of results in different contexts. Both the US and UK researchers will 
be pleased to be contacted with regards the research undertaken with the aim of gaining further insights into the use 
of social capital in higher education. 
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